In order to improve chaos complexity and provide more reliable chaotic system for image encryption, and enhance the security of image encryption algorithm, a new image encryption algorithm based on two-dimensional inverse-trigonometric hyperchaotic system was proposed. Firstly, based on one-dimensional triangular function, a two-dimensional inverse-trigonometric hyperchaotic system was constructed. Compared with some two-dimensional chaotic systems, this system had wider chaotic range, more random iteration sequences and better ergodicity by simulation experiments about bifurcation diagram and Lyapunov exponent. Then based on the proposed chaotic system, the "scrambling-diffusion" strategy was designed and different keys were given, which were used to generate different hyperchaotic sequences. The image matrix was scrambled without repetition by hyperchaotic sequences, then the scrambled sequence were shifted and diffused. So the ciphertext was obtained by looping thrice. Finally, histogram analysis, key space analysis, correlation analysis of adjacent pixels, plaintext sensitivity analysis and information entropy analysis were carried out. The test values of Number of Pixels Change Rate (NPCR) and Unified Average Changing Intersity (UACI) of ciphertext images were very close to their ideal expected values. The test results of information entropy were about 7.997, which was also very close to the expected value of 8. The experimental results show that the image encryption system has more reliable security, stronger ability to resist attacks, and had a good application prospect in the field of image security.